What I did

Many people have given the feedback that they wish Ape supported more sophisticated smart contract testing paradigms, such as Foundry's contract-based testing workflow (using Solidity-based tests). This enables the testing of more complex behaviors like reentrancy, as well as more closely mimics the "look and feel" of being an enduser of such contracts (for example, library developers). Most frameworks have found a way to adopt this style of testing due to it's many benefits (including dramatic improvements in speed from executing in single-txn context within tests), so it's about time Ape supported this.

This is especially important to enable more advanced styles of testing like "fuzz" testing, and property/stateful tests (using Hypothesis ofc), which often catch complex bugs in user smart contracts. While it is possible to use Hypothesis with Ape, the experience of doing so creates a massive slow down. As an example, experiments with stateful tests often took almost an hour to execute. Adopting the contract-based testing paradigm cuts down on test execution time when creating sophisticated testing suites from hours to minutes.

fixes: #363
fixes: #351

How I did it

Ape already has excellent support for compiling smart contracts through it's plugin system, so a smart contract-based testing flow starts with detecting contracts in the tests/ folder to treat as "contract tests". In order to integrate well with other such frameworks (for example foundry, we don't want to co-opt the same tests from that framework, instead encouraging an easy migration path), we skip registering tests using the popular .t.{ext} naming style from our tests folder (Ape already supports "alternative" test folder names e.g. test/).

Secondly, Ape has very nice and broad integration with pytest including support for pytest features like fixtures, so it seems best to leverage that in our contract testing workflow to reduce the work of managing test setup and execution. For example, using pytest fixtures in our contract tests is done simply by looking up fixtures from those already registered when running the test (at that folder level) when a test has an argument that matches. Another example is things like markers, which we decided to support by leveraging custom Natspec declarations in our tests (e.g. @custom:test:mark:xfail {reason}).

Using Natspec turned out to be a very rich and easy way to enable this feature more broadly, so we added support for configuring both Pytest and Hypothesis this way. For example, we allow parametrizing contract tests across a range of values for particular argument(s), configuring hypothesis test settings, checking revert reasons when executing tests, or checking for the existence of contract logs (after executing our tests). Due to the use of Natspec, this works seamlessly out of the box with both Solidity and Vyper languages, meaning that other Ape compiler plugins can easily adapt this workflow by simply supporting custom Natspec-style declarations.

How to verify it

ape test tests/integration/*tests

Write some tests

Checklist

• All changes are completed
• Change is covered in tests
• Documentation is complete